Hi @mudler, friendly ping — this PR adds FunASR as a speech recognition backend for LocalAI. Ready for review whenever convenient. Thanks!

Hi @mudler, friendly follow-up — this PR adds FunASR as a speech recognition backend for LocalAI. FunASR recently added an agent-friendly CLI (funasr audio.wav --output-format json) and supports OpenAI-compatible API. Ready for review. Thanks!

Rebased onto the latest master (resolved the backend/index.yaml conflict — the FunASR and supertonic entries now coexist) and signed off the commit, so DCO passes and the PR is mergeable again. Ready for review whenever you have a chance, @mudler — thanks! 🙏

Rebased this PR onto the latest master and resolved the backend/index.yaml conflict again. The upstream supertonic Metal entries and the FunASR backend entries now coexist in the registry.

Current state:

• PR is mergeable again
• DCO is passing on the refreshed commit
• changed files are still limited to the FunASR backend and backend/index.yaml

Local checks run:

ruby -e 'require "yaml"; data=YAML.load_file("backend/index.yaml"); names=data.select { |x| x.is_a?(Hash) }.map { |x| x["name"] }; abort("missing funasr") unless names.include?("funasr"); abort("missing cpu-funasr") unless names.include?("cpu-funasr"); abort("missing metal-supertonic") unless names.include?("metal-supertonic"); puts "index ok: #{data.length} entries"'
python3 -m py_compile backend/python/funasr/backend.py
rg -n "<<<<<<<|=======|>>>>>>>" backend/index.yaml backend/python/funasr || true
git diff --check upstream/master...HEAD

Quick maintainer-side validation pass after the latest rebase:

• Checked out 8105d38293320388ca18773bffca9507432b7e39 locally.
• git diff --check origin/master...HEAD passes.
• python3 -m py_compile backend/python/funasr/backend.py passes.
• bash -n passes for backend/python/funasr/install.sh, run.sh, and test.sh.
• Parsed all backend/python/funasr/requirements*.txt package lines; pip option lines such as --extra-index-url are present where expected.
• Parsed backend/index.yaml with PyYAML and confirmed the expected 15 FunASR backend names are present, including CPU, CUDA 12/13, ROCm, L4T, and Metal variants.
• make -n -C backend/python/funasr resolves the default target to bash install.sh as expected.

I did not run a full dependency install or model transcription in this bare ops environment, but the registry wiring and backend entrypoint syntax look clean from this pass.

One caveat from a deeper pass so the validation note above is not read as a ready-to-merge sign-off: backend/python/funasr/test.sh currently runs python -m pytest -x -s test.py, but backend/python/funasr/test.py is not part of this PR.

So the registry wiring, syntax, and requirement-file checks are clean, but the make test path still needs either a small backend smoke test or an updated/removed test target before I would call this fully ready from the FunASR side.

Follow-up on the test caveat I noted earlier: I pushed signed head f258f5f4 to add backend/python/funasr/test.py.

What it covers without downloading FunASR weights:

• Health() returns OK.
• LoadModel() uses the default iic/SenseVoiceSmall, fsmn-vad, CPU fallback, and disable_update=True.
• AudioTranscription() forwards a trimmed language parameter to model.generate() and builds transcript text/segments.
• Missing audio input returns an empty transcript without calling the model.

Validation on the refreshed branch:

python3 -m pytest -x -s backend/python/funasr/test.py
# 4 passed
python3 -m py_compile backend/python/funasr/backend.py backend/python/funasr/test.py
bash -n backend/python/funasr/install.sh backend/python/funasr/run.sh backend/python/funasr/test.sh
git diff --check

Note: running bash backend/python/funasr/test.sh directly on my remote ops host still stops before pytest because that host does not have uv installed (../common/libbackend.sh: uv: command not found). The test target itself is now present and passes when invoked with pytest.

Rebased this PR onto current master (d521608e), so it now includes the upstream transcription resolver fix from #10731 (cd65a1f6), including the language / translate handling added there.

Current head is bc412813 on LauraGPT:feat/funasr-backend.

Validation run locally on the rebased branch:

• python3 -m pytest -x -s backend/python/funasr/test.py -> 4 passed
• python3 -m py_compile backend/python/funasr/backend.py backend/python/funasr/test.py
• bash -n backend/python/funasr/install.sh backend/python/funasr/run.sh backend/python/funasr/test.sh
• git diff --check upstream/master...HEAD

GitHub DCO is passing on the new head.

I pulled the failing fork-side tests job log for current head bc412813 (LauraGPT/LocalAI, Actions job 85751160435) so this failure is not read as a FunASR backend regression.

The job reaches the CodeQL/code-scanning upload step and fails because the fork token cannot access the CodeQL API endpoints:

GITHUB_TOKEN Permissions:
  Contents: read
  Metadata: read
  Packages: read
...
This run of the CodeQL Action does not have permission to access the CodeQL Action API endpoints. This could be because the Action is running on a pull request from a fork.
...
Resource not accessible by integration - https://docs.github.com/rest

So the visible failing tests check on the fork is a fork-permission/code-scanning upload gate, not a failure in backend/python/funasr. The upstream DCO check on mudler/LocalAI is still passing for the same head.

For the FunASR backend itself, the latest local validation remains:

python3 -m pytest -x -s backend/python/funasr/test.py  # 4 passed
python3 -m py_compile backend/python/funasr/backend.py backend/python/funasr/test.py
bash -n backend/python/funasr/install.sh backend/python/funasr/run.sh backend/python/funasr/test.sh
git diff --check upstream/master...HEAD

Pushed signed head e7bde3ed after exercising the backend through its real install and inference paths.

This update fixes three runtime gaps found during that pass:

• adds torchaudio to each profile that explicitly installs Torch;
• runs the existing unittest suite through LocalAI's shared runUnittests helper instead of relying on undeclared pytest;
• applies FunASR's rich-transcription postprocessor so SenseVoice language/emotion/event tags do not leak into the OpenAI-compatible transcript.

Verification on ind-gpu8:

• bash test.sh -> 6 passed;
• CPU profile -> official SenseVoiceSmall weights loaded and transcribed a 6-second Chinese sample;
• H100 / CUDA 12 -> the same real-model transcription passed;
• actual gRPC BackendStub -> both LoadModel and AudioTranscription passed;
• all three paths returned 我现在在录一段测试音频 with no rich tags;
• py_compile, bash -n, Ruff, git diff --check, and DCO pass.

The PR description now contains the complete reproducible verification summary. Ready for review.

Follow-up for signed head b970f89:

• merged current master (cdb6702) cleanly; the branch is now 0 commits behind;
• registered FunASR across the required build surfaces: nine backend-matrix profiles, all six top-level Makefile sites, the backend index/capabilities, and backend documentation;
• updated install profiles to current PyTorch indexes (cu130, rocm7.0, Intel XPU, and Jetson L4T) and added Intel XPU device selection plus profile coverage;
• bash test.sh passes all 8 tests; matrix/index consistency, make -n docker-build-funasr, Ruff, py_compile, bash -n, git diff --check, and DCO all pass;
• a fresh CUDA 12 profile install on an H100 passed real protobuf/gRPC health, model load, and SenseVoiceSmall transcription, returning: 欢迎大家来体验达摩院推出的语音识别模型。

The upstream DCO check is green. The other eight workflows currently show action_required with zero jobs, so there is no failing log to address from the contributor side. @mudler, could you approve those workflow runs and review the backend when convenient?

@mudler, could you take a look at this FunASR backend when convenient? The branch is current with master and DCO passes. I have now exercised the actual LocalAI install, unit-test helper, CPU and CUDA SenseVoice inference, and gRPC LoadModel/AudioTranscription paths; all returned clean OpenAI-compatible text, and the six focused tests pass. There are no remaining code-owned CI failures on the PR head.

Exact-head update for d1f6f165 (2026-07-15):

• the upstream merge changed none of the PR-owned FunASR files relative to the previously validated b970f893 tree;
• all 8 backend tests, Ruff, py_compile, shell syntax, matrix/index image mapping, Docker build dry-run, and git diff --check pass;
• a fresh CPU run loaded the official SenseVoiceSmall model through BackendServicer and transcribed the Chinese sample as 开饭时间早上九点至下午五点, with one segment and no leaked rich-transcription tags.